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THE  RELATION  OF  EXPERIMEN TAI.  TESTS  OF  REASONING 
AND  CREATIVE  THINKING  TO  RESEAPCH  PERFORMANCE^ 


In  studies  of  nbihties  considered  to  be  innportAnt  to  the  success  of 
high-level  personnel,  several  types  of  intellectual  activities  are  being  ex- 
plored; reasoning,  creative  thinking,  evaluation,  and  planning.  The  general 
objectives  of  the  project^  have  been  to  isolate  and  define  various  abilities 
underlying  perfornnance  in  each  of  these  domains  by  nneans  of  factor -analy- 
sis techniques  and  to  relate  these  abilities  to  appropriate  aspects  of  per- 
formance of  certain  types  of  high-level  p>ersonnel.  While  major  emphasis 
has  been  directed  toward  psychological  determination  of  intellectual  abili- 
ties, the  scope  of  the  project  investigations  includes  practical  validation  of 
experimental  tests.  Studies  of  industrial  personnel  afford  one  approach  to 
practical  validation  v/hen  adequate  criteria  of  performance  can  be  obtained. 


PURPOSE  AND  PLAN 


The  purpose  of  the  investigation  was  to  obtain  from  an  industrial 
source  evidence  which  would  supplement  the  findings  of  factor-analytic 
studies.  Interest  in  learning  more  about  the  abilities  believed  to  be  mea- 
sured by  the  tests  and  in  determining  which  abilities  might  be  important  in 
certain  research  jobs  prompted  the  investigation.  The  study  was  undertaken 
in  cooperation  v/ith  a large  chemical  manufacturer  that  requested  anonymity. 

The  general  plan  of  the  investigation  '•  ao  to  analyze  the  relationships 
between  performance  on  experimental  tests  of  reasoning  and  creative  think- 
ing and  ratings  of  observable  reasoning  and  creative -thinking  behavior.  Spe- 
cifically, the  plan  called  for  (I)  the  development  of  a rating  scale  describing 
certain  reasoning  and  creative -thinking  traits,  (2)  obtaining  supervisors*  rat- 
ings of  scientistc  on  each  of  the  traits,  and  (3)  relating  performance  on  the 
experimental  tests  to  ratings  on  the  various  abilities. 

It  v'.as  expected  that  there  would  be  some  correspondence  between  res- 
pective tests  of  primary  abilities  and  ratings  of  those  abilities.  .Also,  it  was 
expected  that  tests  representing  abilities  would  relate  positively  to  different 
degrees  to  the  ’'overall*'  ratings  and  rankings  of  creative  thinking  of  individuals. 


^E.M.  Glaser  of  Edward  Glaser  & Associates,  was  instrumental  in  ini- 
tiating th-i  investigation.  We  wish  to  thank  D.  E.  Pye  and  David  Martin  for 
their  cooperation  in  the  planning  and  conduct  of  the  study.  H.  L.  Coderre,  of 
Rohrer,  Hibler  and  Replogle,  kindly  consented  to  rank  the  examinees. 

^Contract  N6onr-23810,  Project  NR  150-044,  with  the  Office  of  Naval 
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Experimental  Materials 


The  Rating  Scale 

The  broad  principle*  of  the  rating  scale  were  devised  in  conference 
among  two  company  executive*,  an  industrial  psychologist  retained  by  the 
company,  and  project  personnel.  Creative -thinking  and  reasoning  abUitie- 
were  discussed  in  their  relation  to  observable  performance  of  ernployees. 

In  this  conference,  eight  creative  - thinking  and  reasoning  abilities  were  se- 
lected and  an  overall-effectiveneso-to-the-company  trait  wac  included. 

The  final  rating  scale  comprising  nine  traits  developed  by  the  project  was 
of  graphic  form  describing  the  abilities  in  nontechnical  language  (bee  Appen- 
dix A). 


The  Tests 


Thirteen  teste  were  included  in  the  test  battery.  Table  1 s'nov'i-  t>»e 
ability  each  test  was  believed  to  measure.  Included  is  the  trait  name  g'ven 
to  the  ability  for  rating  purposes.  Ten  of  the  teste  are  experimental,  having 
been  used  only  in  previous  factor  analyses.  Three  of  the  tests  are  available 
commercially.  Spatial  Visualization  was  included  to  account  for  spatial-vis- 
ualizatron  variance;  Verbal  Comprehension  was  included  to  account  for  ver- 
bal-comprehension variance.  The  Watcon-Glaser  Critical  Thinking  Apprais- 
al was  included  because  of  interest  in  its  relation  to  the  ratings. 


The  Subjects 

The  subjects  made  available  for  the  study  were  research  personnel. 

The  research  conducted  by  the  company  is  primarily  chemical  research  and 
the  employees  are  largely  chemists  and  chemical  engineers.  Hov.-e'/er,  bio- 
logists, physicists,  and  other  scientists  were  also  employed  and  were  includ- 
ed among  the  subjects.  The  group  was  considered  to  exemplify  a creative  in- 
dustrial group  in  that  their  routine  duties  require  solving  problems,  thinking 
up  new  processes  of  manufacture,  and  devising  new  products  and  methods. 

The  possibility  for  observing  creative  activity  should  be  very  favorable  among 
such  a group. 


Operations 


Ratings 

Ratings  of  examinees  were  made  prior  to  the  test  administration.  A pro- 
ject representative  discussed  each  trait  with  the  supervisors  and  pointed  out 
some  of  the  errors  commonly  made  by  raters.  The  supervisors  made  their 
ratings  without  any  apparent  difficulty. 


TABLE  1 


TRAITS.  ABILITIES,  AND  TESTS 


Trait Ability  Teat 

Tdeatlooal  fluency  *1. 

Asaociational  fluency  2. 


A.  Onantib/  of  ideas 

B.  Quality  of  ideas 

C.  Evaluation  of  ideas 

D.  Adaptive  flexibility 

E.  Versatility  of  ideas 

F.  Sensitivity  to 

problems 

G.  Analysis  of  problems 

H.  Improvising 

I.  Overall  effectiveness 
Reference 


Originality  *3. 

Logical  reasoning  4. 

Adaptive  flexibility  5. 

Spontaneous  flexibility  6. 


Sensitivity  to  problems  7. 

General  reasoning  8. 

Eduction  of  conceptual  9. 
relations 

Redefinition  10. 

Eduction  of  correlates  11. 


Spatial  visualization  12. 
Verbal  comprehension  13. 

14. 


Plot  Titles  (low  quality) 
Controlled  Associations 

Plot  Titles  (cleverness) 

Logical  Reasoning 

Match  Problems 

Unusual  Uses 

Apparatus  Test 

Ship  Destination 
Verbal  Analogies  1 

Gestalt  Transformation 
Correlate  Completion 


Spatial  Visualization 
Verbal  Comprehension 

Watsoo'Glaser  Critical 
Thinking  Appraisal 


‘Although  only  13  testa  were  included  in  the  test  battery,  Plot  Titles  is 
scored  for  two  abilities. 
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Test  Administration 


Testing  was  conducted  in  three  four-hour  sessions  during  one  day  and 
the  following  afternoon.  Each  test  group  numbered  approximately  20;  alto- 
gether 64  examinees  were  tested.  Before  each  test  administration,  the  gen- 
eral nature  of  the  investigation  was  explained  to  the  group  and  the  examinees 
were  told  that  they  would  be  informed  of  their  scores.  A biographical  data 
blank  was  completed  by  each  examinee  during  the  test  admlnlsl/ation  (see 
Appendix  B). 


RESULTS 

Description  of  the  Sample 


Table  2 describes  the  sample  according  to  age,  sex,  length  of  employ- 
ment and  other  characteristics.  It  will  be  noticed  that  the  sample  is  much 
above  the  average  in  education.  All  of  the  group  were  high  school  graduates; 
only  three  people  did  not  have  a bachelor's  degree  and  31  of  the  group  of  64 
had  attended  graduate  school.  Twelve  doctor's  and  11  master's  degrees  were 
held  by  the  group.  The  group  was  relatively  young,  and  the  majority  of  the 
sample  had  worked  for  the  company  less  than  three  years. 


Obtained  Data 


Ratings 

Thirteen  supervisors  rated  the  examinees.  However,  every  rr.ter  did  not 
rate  every  examinee.  Most  of  the  raters  were  examinees  who  rated  only  those 
examinees  they  knew  well  enough  to  rate,  not  including  themselves.  Few  exam- 
inees were  rated  in  common  by  even  three  or  four  raters,  and  many  were  rated 
by  only  one. 

Although  every  examinee  had  been  rated,  it  was  necessary  to  select  only 
those  cases  in  which  certain  statistical  requirements  could  be  met.  For  exam- 
ple, it  was  felt  desirable  to  equate  all  of  the  ratings  to  a common  mean  and  stan- 
dard deviation.  A standard  deviation  computed  on  a small  N is  likely  to  be  un- 
reliable; consequently,  the  ratings  of  raters  who  had  rated  only  a few  examinees 
were  omitted.  Five  raters  had  rated  between  13  and  21  examinees  each,  or  a 
total  of  53  examinees.  This  sample  of  53  was  considered  for  farther  study,  and 
the  ratings  of  four  of  the  raters  were  adjusted  to  the  mean  and  standard  devia- 
tion of  the  fifth  rater  who  had  rated  21  examinees. 


Reliabilities 


Relisbilities  of  ratings  could  not  be  computed  in  the  regular  manner  be- 
cause the  examinees  were  not  rated  in  common,  uniformly,  by  the  five  raters. 


TABLE  2 


DESCRIPTION  OF  THE  SAMPLE  (N*64) 


1.  Sex: 


Male 

Female 


Total  = 


II.  Age : 

Years 

48.0  - 51.9 

44.0  - 47.9 
40.6  - 43.9 

36.0  - 39.9 

32.0  - 35.9 

28.0  - 31.9 

24.0  - 27.9 

20.0  - 23.9 

Total  = 


III.  Major  Fields: 


Bio.  Science 

Botany 

Chem.  Engng 

Chemistry 

Electr.  Engng 

Entomology 

Metallurgy 

Nuclear  Chem. 

Physical  Chem. 

Physical  Science 

Physics 

Plant  Pathology 
ToUl  s 


IV.  Education: 


A.  College 

9 

Years 

64 

3.1  - 4.0 

63 

2.1  - 3.0 

0 

1.1  - 2.0 

1 

0.1  - 1.0 

_0 

Total  = 

64 

1 

5 

B.  Graduate  School 

4 

0 

2.1  - 3.0 

13 

Q 

1.1  - 2.0 

7 

17 

0.1  > 1.0 

n. 

11 

Total  = 

31 

8 

64 

C.  Highest  degree  attained: 

B.A. 

12 

B.S. 

27 

f 

M.A. 

1 

M.S. 

10 

1 

Ph.D. 

12 

1 

None 

2 

17 

37 

1 

ToUl  = 

64 

i 

1 

V.  Leofith  of  employment  at  company. 

1 

1 

Years 

J. 

8 

18.0  - 19.9 

2 

1 

16.0  - 17.9 

2 

1 

14.0  - 15.9 

1 

1 

12.0  - 13.9 

5 

10.0  - 11.9 

5 

71 

8.0  - 9.9 

1 

6.0  - 7.9 

6 

4.0  - 5.9 

7 

2.0  - 3.9 

11 

0.0  - 1.9 

a 

Total  s 

64 

*Soma  examioaes  reported  more  than  one  major  field. 
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To  estimate  the  reliability  of  ratings,  an  atialysis-of-variance  procedure  was 
used.  The  within- variance  was  divided  by  the  total  variance,  and  the  quotient 
was  subtracted  from  one.  However,  the  reliabilities  computed  in  this  way  are 
estimates  of  individual- rating  reliabilities.  As  each  examinee  was,  on  the  aver- 
age. rated  by  two  and  one-half  raters,  an  estimate  of  the  reliability  of  the  com- 
posite rating  was  computed  using  a correction  factor  of  2.5  In  the  Spearman- 
Brown  formula.  Table  3 shows  both  the  individual- rating  reliabilities  and  the 
composite -rating  reliabilities.  Ratings  for  trait  D,  adaptive  flexibility,  have  the 
lowest  reliability,  and  for  trait  I,  overall  effectiveness,  the  highest. 


Inter  cor  relations 


Table  4 shows  the  correlation  based  on  average  ratings  of  every  trait  v/ith 
every  other  trait.  Trait  D.  adaptive  flexibility,  has  the  lowest  average  intercor- 
relation with  the  remaining  traits,  and  trait  I,  overall  effectiveness,  has  the  high- 
est average  intercorrelation.  The  correlation  of  trait  I with  the  other  traits  is 
spurious  to  the  extent  that  the  description  of  this  trait,  "...this  trait  is  made  up 
of  all  the  previous  traits,  but  in  addition..."  effected  overlap. 


Tests 


Scoring.  The  tests  were  scored  by  hand  and  accuracy  of  scoring  was 
checked.  In  one  test  administration,  only  35  minutes  was  allowed  for  the  Wat- 
son-Glaser  Critical  Thinking  Appraisal  instead  of  the  customary  45-50  min- 
utes. As  the  test  is  composed  of  five  untimed  subtests,  it  was  decided  to  use 
only  the  scores  on  subtests  completed  by  every  examinee.  The  first  two  sob- 
tests.  Inferences  and  Assumptions,  were  completed  by  everyone,  and  these  two 
were  included  for  further  study. 

Reliabilities.  Reliabilities,  based  on  the  sample  of  64,  were  computed 
for  all  tests.  Odd-even  or  alternate -form  estimates  were  obtained  and  cor- 
rected for  length.  The  reliabilities  are  shown  in  Table  5. 


Relationships 


In  determining  the  relationships  between  tests  and  ratings,  two  proce- 
dures could  be  followed.  By  one  method,  ratings  would  be  averaged  and  then 
correlations  between  average  ratings  and  test  scores  would  be  computed.  In 
the  second  method,  correlatioos  would  be  computed  for  individual  raters  and 
then  their  individual  trait-test  correlations  averaged.  The  first  method  would 
require  computing  135  correlations  (9  traits  x 15  tests).  The  second  method 
would  require  computation  of  675  correlations  (9  traits  x 15  tests  x 5 raters). 


Correlations  between  Tests  and  Averaged  Ratings 

In  the  interests  of  economy,  it  was  decided  to  compute  correlations  be- 
tween test  scores  and  averaged  ratings  for  the  sample  of  53.  These  correla- 
tions are  shown  in  Table  6.  Seventeen  of  the  correlations  are  significant  at 
the  five  per  cent  level  of  probability  and  three  are  statistically  significant  at 
the  one  per  cent  level. 
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TABLE  3 

ESTIMATES  OF  RATED-TRAIT  RELIABILITIES 


Traits 

Corrected 

A. 

Quantity  of  ideas 

.40 

.62 

B. 

Quality  of  ideas 

.43 

.65 

C. 

Evaluation  of  ideas 

.26 

.47 

D, 

Adaptive  flexibility 

.31 

.53 

E. 

Versatility  of  ideas 

.37 

.59 

F. 

Sensitivity  to  problems 

.42 

.64 

C. 

Analysis  of  problems 

.42 

.64 

H. 

Improvising 

.47 

.69 

I . 

Overall  effectiveness 

.68 

.84 

a 2 . within  variance 
total  variance 

**Corrected  by  2.5  with  Spearman- Brown  formula 


TABLE  4 

INTER  CORRELATION  OF  RATED  TRAITS 


A 

B 

C D 

E F G H I 

A. 

Quantity  of  ideas 

B. 

Quality  of  ideas 

.71 

C. 

Evaluation  of  ideas 

.60 

.83 

D. 

Adaptive  flexibility 

.76 

.62 

.59 

E. 

Versatility  of  ideas 

.87 

.71 

.69  .73 

F. 

Sensitivity  to  problems 

.81 

.78 

.79  .67 

.79 

G. 

Analysis  of  problems 

.65 

.82 

.89  .65 

.69  .78 

H. 

Improvising 

.69 

.72 

.74  .64 

.75  .72  .80 

1 . 

Overall  effectiveness 

.72 

.86 

.89  .66 

.78  .84  .88  .80 

Mr 


.73  .76  .75  .66  .75  .77  .77  .73  .80 


V 


• • 1 


Ntv  ' 


TABLE  6 

RELIABILITIES  OF  TESTS 


Te  sti 

Reliabilities 

Est.” 

Plot  Titles  {low  quality) 

.83 

AF 

Controlled  Associations 

.86 

OE 

Plot  Titles  (Cleverness) 

.73 

AF 

Logical  Reasoning 

.77 

AF 

Match  Problems 

.70 

AF 

Unusual  Uses 

.66 

AF 

Apoaratus  Test 

.78 

AF 

Ship  Destination 

.84 

OE 

Verbal  Analogies  I 

.45 

AF 

Gestalt  Transformation 

.29 

AF 

Correlate  Completion 

.70 

OE 

Spatial  Visualization 

.96 

OE 

Verbal  Comprehension 

.91 

OE 

Watson-Glaser  Inferences 

.41 

OE 

Watson-Glaser  Assumptions 

.80 

OE 

*AF  s alternatc'form  relimbilities; 
OC  = odd-eveo  reliabilities. 

All  estimates  are  corrected  Cor  length. 
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TABLE  6 


CORRELATIONS  BETWEEN  TESTS  AND  AVERAGED  RATINGS 


Test Ratings 


Qn 

Ql 

Ev 

A£ 

Vs 

Sn 

An 

Im 

Cm® 

Plot  Titles  (low  quality) 

.00  - 

.02 

.08 

.04 

.06 

.08 

.07 

.25 

.11 

Controlled  Association 

.21 

.20 

.24 

.40* 

.20 

.28*  .28* 

.28* 

.32* 

Plot  Titles  (clevernees) 

.20 

.04 

.15 

.11 

.21 

.13 

.18 

.16 

.13 

Logical  Reasoning 

-.01 

.06 

.08 

.12  - 

.15  - 

.13 

.07 

.21 

-.02 

Match  Problems 

.23 

.28* 

.20 

.23 

.09 

.31*  .16 

.08 

.21 

Unusual  Uses 

.13 

.09 

.12 

.32* 

.28* 

.18 

.05 

.12 

.22 

Apparatus  Test 

.07  - 

.01 

.19 

.17 

.13 

.13 

.12 

.08 

.16 

Ship  Destination 

.29* 

* 

o 

.25 

.27 

.17 

.24 

,22 

.17 

.26 

Verbal  Analogies  I 

.17 

.22 

.31* 

.28* 

-.05 

.20 

.26 

.10 

.26 

Gestalt  Transformation 

.26 

.16 

.17 

.30* 

.24 

.17 

.02 

.24 

.20 

Correlate  Completion 

.05 

.16 

.17 

.16  • 

.02 

.07 

.11  - 

.16 

.19 

Spatial  Visualization 

.16 

.24 

.20 

.24 

.07 

.11 

.22 

.02 

.19 

Verbal  Comprehension 

.20 

.28* 

.47* 

.22 

.18 

.28*  .31* 

.34* 

.35* 

Watson-Glaser  Inferences 

.06 

.13 

.05 

.07 

.04 

.02 

.05 

.07 

.03 

Watson-Glaser  Assumptions 

.01  - 

.04 

.11 

.08  - 

.02  - 

.16 

.17  - 

.03 

.03 

*For  50  df’«:  r s ,273  ia  significant  at  5%  level. 

r = .354  ia  aignificant  at  19^  level. 


^Abbrevlationa  are  as  foUov/s: 


Qn  B Quantity 

Ql  B Quality 

Ev  B Evaluation 

A£  s Adaptive  flexibility 


Va  B Versatility 
Sn  B Sensitivity 
An  B Analysis 
Im  « Improvising 


Cm  B Composite 
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Verbal  Comprehension  had  significant  correlations  with  six  of  the  nine 
traits,  two  correlations  being  significant  beyond  the  one  per  cent  level.  Con- 
trolled Associations  had  five  significant  correlations  with  the  traits.  Match 
Problems,  Unusual  Usesi  Verbal  Analogies  I.  and  Ship  Destination  each  had 
two  significant  correlations  with  the  traits.  Gestalt  Transformation  had  one 
significant  correlation.  Eight  testa  had  no  significant  correlations  with  the 
traits. 


Examining  the  relationships  from  the  standpoint  of  the  traits,  it  will  be 
noticed  that  every  trait  has  at  least  one  significant  correlation  with  a test. 
Trait  D,  adaptive  flexibility,  has  significant  correlations  with  four  tests. 
Trait  F,  sensitivity  to  problems,  and  trait  B,  quality  of  ideas,  each  has  three 
significant  correlations  with  tests.  The  remaining  traits  have  one  or  two  sig- 
nificant correlations  with  the  tests. 

These  results  yield  little  information  concerning  the  factorial  nature  of 
the  tests,  although  the  indications  are  that  the  most  verbally  fluent  person  is 
the  person  most  apt  to  obtain  high  ratings  on  the  traits. 


Average  Correlations  Between  Tests  and  Individual  Ratings 

Averaging  ratings  before  computing  correlations  between  tests  and 
traits  may  have  obscured  individual  differences  among  raters.  In  order  to 
examine  this  possibility,  correlations  were  computed  between  individual- 
rater  ratings  and  test  scores  for  selected  testa.  Table  7 shows  these  cor- 
relations  and  also  averages  of  them. 

The  results  from  this  phase  of  the  investigation  shows  that  each  rater 
was  rating  largely  on  the  basis  of  one  or  two  traits.  Evidence  that  these 
traits  were  not  the  same  for  all  raters  is  found  in  Table  7 in  which  a certain 
test  tends  to  correlate  substantially  with  all  the  traits  for  a particular  rater 
but  correlates  low  with  the  traits  for  a different  rater.  For  example,  one  rat- 
er appeared  to  be  rating  primarily  on  a logical  reasoning  ability,  a second  ap- 
parently rated  on  the  basis  of  originality,  and  a third  on  the  basis  of  the  rede- 
finition ability. 

Combining  the  positive  and  negative  individual  rater  correlations  tended 
to  produce  an  average  correlation  that  was  statistically  insignificant.  It  seems 
evident  that  little  confidence  can  be  placed  in  the  ratings  as  criteria  for  deter- 
mining what  the  tests  are  measuring. 


Correlations  Between  Rank  Order  and  Tests 

Three  supervisors  and  an  industrial  psychologist  were  asked  to  rank  the 
employees  they  knew  well  enough  to  rank  on  a continuum  of  creativity.  That  is. 
the  employees  were  placed  in  rank  order  from  the  most  creative  to  the  least 
creative.  Product-moment  correlations  were  computed  between  these  rankings 
and  test  scores. 
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TABLE  7 

INDIVIDUAL-RATER  CORRELATIONS 


Logical  Reasoning 


Trait 

T 

U 

V 

Y 

Average^ 

A 

-.17 

-.20 

.14 

-.23 

.40  ' 

.12 

B 

-.27 

.07 

.18 

-.31 

.63 

.08 

C 

-.33 

-.19 

.30 

-.05 

.52 

.06 

D 

-.12 

.14 

.52 

-.31 

.75 

.25 

E 

-.17 

-.27 

.13 

-.43 

.18 

-.12 

F 

-.32 

-.25 

.26 

-.36 

.35 

-.07 

G 

-.40 

-.16 

.35 

-.27 

.73 

.09 

H 

-.31 

.17 

.28 

-.15 

.58 

.13 

1 

-.29 

-.22 

.28 

-.36 

.52 

-.06 

-.26 

-.09 

.27 

-.26 

.54 

.07 

Gestalt 

Transformation 

A 

.16 

.07 

.18 

.56 

.23 

.24 

B 

.03 

.14 

.28 

.20 

-.01 

.13 

C 

.05 

.32 

.21 

.09 

.15 

.17 

D 

.19 

.28 

.51 

.57 

.13 

.35 

E 

.12 

.14 

.44 

.09 

.17 

.20 

F 

-.28 

.32 

.56 

.41 

-.13 

.20 

G 

-.03 

.34 

.47 

.27 

.09 

.25 

H 

.00 

.19 

.41 

.38 

-.14 

.18 

1 

-.13 

.31 

.39 

.33 

.13 

.21 

Mr 

.01 

.24 

.39 

.34 

.07 

.22 

Plot  Titles  (clevernest 

i 

A 

.44 

.29 

.33 

-.09 

.42 

.30 

B 

.46 

.00 

.12 

-.04 

.09 

.10 

C 

.44 

.18 

.08 

-.05 

.21 

.29 

D 

.50 

.12 

.24 

-.32 

.25 

.17 

E 

.55 

.20 

.22 

-.02 

.45 

.32 

F 

.33 

.20 

.13 

-.19 

.18 

.13 

G 

.33 

.18 

.05 

-.02 

.03 

.12 

H 

.44 

.32 

-.23 

.88 

-.02 

.27 

1 

.29 

.20 

.19 

.26 

.17 

.23 

Mr 

.42 

.19 

.13 

-.02 

.19 

.22 

*Rater  T had  / 

an  N of  21 

; rater 

U.  14; 

V.  14; 

Y.  21;  Z.  17. 

CorielaUoQS  were  converted  into  correapondttig  F4iher’» 
r.  coefficient  before  averaging. 
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Table  8 shows  the  correlations  £or  each  ranker.  Again  it  is  apparent 
that  there  is  some  variation  between  rankers,  although  the  correlations  of 
two  of  the  supervisors  most  actively  associated  with  the  examinees  were  in 
substantial  agreement.  Averages  of  these  two  supervisor's  (Y  and  Z)  rank- 
ings are  also  shown  in  Table  8.  Four  of  the  average  correlations  approach 
statistical  significant  and  represent  relationships  between  rankings  and  gen* 
eral  reasoning,  eduction  of  correlates,  originality,  and  sensitivity  to  prob- 
lems. The  tests  for  these  abilities  are.  respectively.  Ship  Destination,  Cor- 
relate Completion,  Plot  Titles  (cleverness)  and  Apparatus  Test. 

It  will  be  noticed  that  these  results  do  not  substantiate  the  rec  jUs  ob- 
tained with  averaged  ratings  in  which  verbal  facility  appeared  to  be  related  to 
ratings  of  performance. 


CONCLUSIONS  AND  RECOMMENDATIONS 


1.  Ratings  could  not  be  used  as  unique  indicators  of  the  factors  since 
their  intercorrelations  were  much  too  high  relative  to  their  reliabilities.  It 
was  therefore  impossible  to  validate  the  factor  measures  against  ratings  as 
criteria. 

2.  A number  of  significant  validity  coefficients  were  found  between 
tests  and  average  ratings,  but  these  were  low  and  not  substantiated  by  evi- 
dence from  correlations  between  tests  and  rankings. 

3.  When  individual  rater's  ratings  were  correlated  with  tests  it  ap- 
peared that  each  rater  tended  to  emphasize  one  or  two  of  the  factors  mea- 
sured by  the  tests,  regardless  of  what  trait  he  was  rating.  Each  rater  seem- 
ed to  emphasize  a different  factor  or  factors.  Thus,  averaging  ratings  tends 
to  obscure  the  meaning  of  the  validity  coefficients  obtained. 

4.  Rankings  of  the  examinees  for  creativity  were  significantly  (but  not 
very  significantly)  correlated  with  two  creative -thinking  and  two  reasoning 
tests.  None  of  these  four  correlations  were  significant  when  average  ratings 
were  used  as  criteria. 

5.  On  the  basis  of  these  inconclusive  results,  we  cannot  recommend  any 
of  these  tests  for  predicting  performance  of  research  personnel. 

6.  It  is  recommended  in  future  research  of  this  kind  that  consideratly 
more  time  and  attention  be  paid  to  obtaining  analytiref  evaliiatioo*!  of  rerparrh 
performance. 
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TABLE  8 

CORRELATIONS  BETWEEN  RANK  ORDER  AND  TEST  SCORES 


Rankers 

Tests W X Y Z Averages^ 


Plot  Titles  (low  quality) 

-.13 

-.08 

-.06 

.02 

-.02 

Controlled  Associations 

.25 

.19 

.39 

.05 

.23 

Plot  Titles  (cleverness) 

.15 

.13 

.31 

.32 

.32 

Logical  Reasoning 

.23 

-.13 

.12 

.43 

.28 

Match  Problems 

.36 

.09 

.22 

.28 

.26 

Unusual  Uses 

.15 

.15 

.34 

.26 

.28 

Apparatus  Test 

.03 

.11 

.33 

.43 

.38 

Ship  Destination 

.28 

.05 

.28 

.37 

.33 

Verbal  Analogies  1 

.15 

-.06 

.07 

.12 

.10 

Gestalt  Transformation 

.11 

.36 

.10 

-.04 

.03 

Correlate  Completion 

.25 

.-33 

.38 

.27 

.33 

Spatial  Visualization 

.22 

-.03 

.11 

.18 

.14 

Verbal  Comprehension 

.07 

.18 

.11 

.00 

.06 

Watson-Glaser  Inferences 

.10 

-.15 

-.33 

-.06 

-.20 

Watson-Glaser  Assumptions 

.00 

-.10 

-.15 

-.04 

-.10 

^Average  correlations  for  Y and  Z.  Correlations  were  converted  to  corres 
ponding  Fisher's  z coefficients  before  averaging. 


W 

X 

Y 

Z 


Correlations  Between  Rankers 
W X Y Z^ 

.28  .23  .23 

.41*  .41* 

.44* 


^Based  on  12  examinees  ranked  in  cont' 
mon;  others  based  on  22  ranked  in  common. 
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INSTRUCTIONS  TO  RATERS 


Name  of  Rater 


In  connection  with  a research  investigation  in  which  Co.m^any  > is  p?rUci* 
paling,  you  are  asked  to  give  your  opinions  on  several  differed  t traits  of  men  v/hc 
work  under  your  supervision,  either  directly  or  indirectly.  C*u  the  sui  ceeding 
pages  each  trait  will  be  described;  for  each  person  you  know  well  eooL  7U  to  rale 
you  will  be  asked  to  place  a mark  on  the  line  that,  in  your  opiriii.o»  best  ’eEcribe.^ 
him. 


Your  rating  will  be  most  helpiui  it  you  will  follow  tnese  rules  v/hen  mak  ng 
your  ratings . 


1.  Consider  each  trait  separately. 

2.  Consider  each  man's  relative  position  among  all 
research  personnel  you  have  known. 

3.  Avoid  giving  ties. 

Rate  the  entire  group  on  one  trait  before  proceeding  to  the  next  trait.  It 
should  be  emphasized  that  a person  is  likely  to  be  high  on  sor-.e  of  the  trails  and 
low  on  others.  Tt  is  not  likely  that  an  individual  will  be  high  cn  all  of  the  traits 
or  low  on  all  of  them,  although  this  is  possible. 

Concerning  the  second  point,  it  should  be  noted  that  these  ratings  are  rela- 
tive to  all  similar  research  personnel  you  have  known.  The  -.ndividual's  sUno.ing 
in  comparison  to  the  research  personnel  you  have  known  should  determine 
position  on  this  scale.  For  example,  does  the  person  being  rated  compare  favor- 
ably with  the  highest  10%,  is  he  just  average,  or,  as  the  cas  ; may  be,  is  he,  in 
your  opinion,  in  one  of  the  low  categories.  Unless  you  happ»*n  to  be  rating  a group 
that  is  uniformly  good  or  poor,  some  of  the  people  will  be  h.gh  and  some  will  be 
low  on  each  of  the  traits.  The  percentages  along  the  side  of  the  lines  indicate 
the  approximate  distribution  of  ratings  for  a typical  group.  For  example,  ap- 
proximately 10%  will  be  in  the  highest  category  and  10%  will  be  in  the  lowest. 
Twenty -five  per  cent  will  be  in  the  second  highest  and  in  the  next  to  the  lowest 
categories.  Thirty  per  cent  of  such  a group  will  be  in  the  middle  category. 

Finally,  avoid  rating  two  people  at  exactly  the  same  level  if  possible. 

Within  a category,  a mark  can  be  placed  at  the  upper  or  lower  end  or  at  any 
other  points  to  indicate  the  person’s  position  in  the  category.  For  example,  if 
the  person  has  been  Judged  to  be  in  the  highest  10%  of  the  group  he  may  be 
placed  at  the  top  of  this  category,  in  the  middle  portion  of  it,  or  at  the  lower 
end.  An  illustrative  example  of  the  rating  procedure  is  on  the  next  page. 

These  ratings  will  in  no  way  affect  the  standing  of  any  employee.  They 
are  tor  experimental  purposes  only.  Company  X will  receive  an  analysis  of  the 
investigation  but  no  individual  will  be  identified  with  a rating  or  test  score. 
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Trait  A:  Quaotitv  of  idaaa 


Definition:  When  this  person  ie  faced  witit  a queation  or  problem  he  immediately 
has  a lot  of  ideas  about  it.  It  does  not  matter  for  this  trait  whether  the 
ideas  are  any  good  or  not.  It  is  the  number  of  ideas  per  unit  of  time 
that  counts. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


Trait  B: 


Quality  of  idaat 

Definition:  This  person  usually  has  very  good  ideas  to  offer.  He  has  often  made 
suggestions  of  real  value  to  the  company  and  many  of  his  ideas  have 
proven  to  be  major  contributions.  When  approached  for  a suggestion 
or  idea  for  solving  a problem*  this  person  can  usually  come  up  with 
a good  one. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


J. 
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Trait  C: 


Evaluatiop  ot  idaaa 


Definition:  This  person  it  able  to  Judge  the  value  of  his  ideas  rather  accurately. 

He  does  not  suggest  things  which  on  close  examination  prove  to  be 
hasty  opinions  or  ill-considered  suggestions.  Not  all  of  his  ideas  are 
necessarily  good  ones.  but.  on  the  omer  hand,  they  are  usually  careful- 
ly  Judged.  He  has  shown  ability,  not  only  to  evaluate  his  own  ideas,  but 
also  to  Judge  the  worth  of  other  people's  ideas. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


Trait  D:  Adaptive  flextbility 


Definition:  This  person  can  adapt  his  work  to  meet  changing  conditions:  he  is  very 
open  to  new  ideas.  If  a method  or  technique  he  is  using  fails  to  bring 
about  the  desired  result,  he  is  able  to  take  a new  approach  in  an  attempt 
to  obtain  a solution.  He  does  not  uofruitfully  pursue  the  same  approach 
of  a problem  over  long  periods  of  time.  In  a situation  where  achieving 
a very  new  mental  outlook  is  required  before  the  solution  to  the  problem 
is  seen,  this  person  is  more  likely  than  the  average  person  to  achieve 
this  outlook. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


s 
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Trait  E:  Varaatility  of  Ideaa 

Definition;  Thia  peraon  haa  many  highly  diveraifiad  interaata  and  can  offer  helpful 
auggeationa  and  ideaa  on  a number  of  different  aubjects.  He  can  auggeat 
aolutiona  to  problema  outaide  the  field  of  hia  apecialisatlon.  He  haa  uae* 
ful  knowledge  on  a number  of  aeemingly  unrelated  topica  and  could  work 
effectively  in  any  one  of  aeveral  different  fielda.  He  ia  not  narrowly  ape* 
cialiaed  in  hia  interaata  and  thinking,  nor  doea  he  become  *‘loat*‘  when  he 
geta  outaide  of  hia  field. 


Higheat  10% 


Next  25% 


Middle  30% 


Next  25% 


Loweat  10% 


>•% 
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Trait  F:  SengittvU 


Definition:  Thif  person  ia  able  to  point  out  the  existence  of  problems,  either  theo* 
retical  or  practical,  that  would  ordinarily  go  unnoticed.  He  has  often 
shown  the  significance  of  a factor  overlooked  or  dismissed  as  trivial 
by  others.  He  is  aware  of  needs  for  changes  or  for  new  devices  and 
has  seen  defects  or  deficiencies  in  things  as  they  exist.  He  may  or 
may  not  suggest  improvements  for  the  problems  he  has  pointed  out. 
The  important  thing  in  this  trait  is  the  ability  to  recognise  that  a 
problem  exists,  whether  corrective  action  is  suggested  or  not. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


t #• 


Trait  G:  Aaalvata  of  probUma 

Oaflnitioo:  Thia  parson  is  able  to  radaca  a problem  successfully  into  its  fundamen- 
tal components.  He  can  separate  the  more  basic  factors  from  the  irrele- 
vant or  leas  fundamental  ones.  He  is  able  to  formulate  clearly  tbe  prob- 
lem at  hand  by  stating  what  the  important  features  are.  When  faced  with 
a complex  mass  of  data,  this  person  is  usually  able  to  formulate  tbe  prob- 
lem clearly  and  simply.  Tbe  person  who  is  low  on  this  trait  cannot  see 
the  relationships  between  different  factors  of  a problem.  He  is  unable  to 
distinguish  between  the  more  important  and  the  less  important.  He  fre- 
quently “misses  the  point*  in  his  explanations  of  problems. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


Trait  H:  Improviitni 


Definition:  This  person  is  able  to  adapt  material,  methods,  or  techniques  to  meet 
the  demands  of  his  particular  problem.  He  is  able  to  improvise  with 
equipment  and  to  adapt  methods  of  one  procedure  to  meet  the  require* 
ments  of  another.  He  can  see  ways  of  using  things  that  are  different 
from  the  most  common  use.  tf  the  proper  equipment  is  not  available, 
this  person  is  rarely  at  a loss  as  to  how  to  proceed.  He  will  devise  a 
satisfactory  procedure  with  whatever  equipment  is  at  hand. 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


know  well  enough  to  rarte.- 


S 
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Trait  I;  Overall  ef£ectjvene»« 

Definition:  This  trait  is  compoaed,  in  part,  of  all  the  previously  mentioned  traits. 

but.  in  addition,  includes  any  other  qualities  that  should  be  considered. 
Considering  everything  you  believe  to  be  important,  how  much  of  a con- 
tribution does  this  man  make  to  his  organization.  If  you  were  hiring  men 
to  staff  a research  group,  how  well  would  he  "stack  up"  with  the  others? 


a 


Highest  10% 


Next  25% 


Middle  30% 


Next  25% 


Lowest  10% 


